State determination device, state determination method and recording medium

ABSTRACT

A state determination device according to the present invention includes: a memory; and at least one processor coupled to the memory. The processor performs operations. The operations including: combining output from a plurality of determination models for an input image of a road, the plurality of determination models each learned using training data in which at least one of a state of a moving object mounting an imaging device that acquires an image of the road, a state of the road, and an external environment is different.

TECHNICAL FIELD

The present invention relates to processing of information related to aroad, and in particular, to a determination of a state of a road.

BACKGROUND ART

Management of the road responsible for local governments and the likerequires a lot of effort and cost. Therefore, a device that collectsroad information has been proposed (see, for example, PTL 1).

The road measurement device described in PTL 1 determines deteriorationof a road using image data of a surface of the road to output a map inwhich the deterioration and a presentation value of a reward are mapped.The road measurement device described in PTL 1 uses a determinationmodel using image data as an input for a determination of deterioration.This determination model is constructed by a machine learning method.

CITATION LIST Patent Literature

PTL 1: JP 2019-079166 A

SUMMARY OF INVENTION Technical Problem

The capturing of the image of the road is outdoor imaging and imaging invarious weather and time. Therefore, the captured image is an imagecaptured under various imaging conditions.

When an image captured under various imaging conditions is to bedetermined using one determination model, it is difficult to ensure theaccuracy of determination.

The reason will be described using weather as an example of the imagingcondition.

The determination model executes learning before determination.

For example, a determination model learned using an image at the time offine weather can appropriately determine an image at the time of fineweather. However, the determination model learned using the image at thetime of fine weather cannot always appropriately determine the imagecaptured at the time of cloudy weather and rainy weather.

Alternatively, the determination model learned using a plurality ofweather images such as fine weather, cloudy weather, and rainy weathercan be determined with a certain degree of accuracy with respect to theimage at the time of each weather. However, even if using such adetermination model, it is difficult to improve the accuracy ofdetermination for images at the time of all weathers.

As described above, in a case where images at the time of all weathersare determined using one determination model, it is difficult to improvethe accuracy of determination.

The technique described in PTL 1 uses one determination model.Therefore, the technique described in PTL 1 has an issue that it isdifficult to improve the accuracy of a determination of the stateregarding the road.

An object of the present invention is to provide a state determinationdevice and the like that solve the above issue and improve the accuracyof a determination of a state regarding a road.

Solution to Problem

According to an aspect of the present invention, a state determinationdevice includes:

-   -   a plurality of determination models each learned using training        data in which at least one of a state of a moving object        mounting an imaging device that acquires an image of a road, a        state of the road, and an external environment is different; and    -   an output combining means configured to combine output from the        plurality of determination models for an input image of the        road.

According to an aspect of the present invention, a state determinationsystem includes:

-   -   the state determination device described above;    -   an imaging device that acquires an image of a road to be        determined to output the image to the state determination        device, and/or a storage device that stores the image of the        road to be determined; and    -   a display device that displays a state of the road output by the        state determination device.

According to an aspect of the present invention, a state determinationmethod includes:

-   -   by a state determination device including a plurality of        determination models each learned using training data in which        at least one of a state of a moving object mounting an imaging        device that acquires an image of a road, a state of the road,        and an external environment is different,    -   combining output from the plurality of determination models for        an input image of a road.

According to an aspect of the present invention, a state determinationmethod includes:

-   -   by a state determination device, executing the state        determination method described above,    -   by an imaging device, acquiring an image to be determined to        output the image to the state determination device, and/or, by        the storage device, storing the image to be determined; and    -   by a display device, displaying a state of a road output by the        state determination device.

According to an aspect of the present invention, a recording mediumrecords a program for causing a computer, the computer including aplurality of determination models each learned using training data inwhich at least one of a state of a moving object mounting an imagingdevice that acquires an image of a road, a state of the road, and anexternal environment is different, to execute:

-   -   a processing of combining output from the plurality of        determination models for an input image of the road.

Advantageous Effects of Invention

An example advantage according to the present invention is to improvethe accuracy of a determination of a state related to a road.

BRIEF DESCRIPTION OF DRAWINGS

FIG. 1 is a block diagram illustrating an example of a configuration ofa state determination device according to a first example embodiment.

FIG. 2 is a diagram illustrating an example of a weight.

FIG. 3 is a flowchart illustrating an example of operation in a learningphase of a determination model.

FIG. 4 is a flowchart illustrating an example of an operation in adetermination phase of the state determination device according to thefirst example embodiment.

FIG. 5 is a block diagram illustrating an example of a hardwareconfiguration of the state determination device.

FIG. 6 is a block diagram illustrating an example of a configuration ofa state determination system including the state determination deviceaccording to the first example embodiment.

FIG. 7 is a block diagram illustrating an example of a configuration ofa state determination device according to a second example embodiment.

FIG. 8 is a flowchart illustrating an example of an operation in adetermination phase of the state determination device according to thesecond example embodiment.

FIG. 9 is a block diagram illustrating an example of a configuration ofa state determination device according to a third example embodiment.

FIG. 10 is a flowchart illustrating an example of operation in alearning phase of a weight determination unit according to the thirdexample embodiment.

FIG. 11 is a block diagram illustrating an example of a configuration ofa state determination device according to a fourth example embodiment.

FIG. 12 is a flowchart illustrating an example of operation in alearning phase of a weight determination unit according to the fourthexample embodiment.

FIG. 13 is a block diagram illustrating an example of a configuration ofa state determination system including the state determination deviceaccording to the fourth example embodiment.

FIG. 14 is a block diagram illustrating an example of a configuration ofa state determination device according to a fifth example embodiment.

FIG. 15 is a view illustrating an example in a case where a plurality ofdrive recorders is used as an imaging device.

FIG. 16 is a view illustrating an example of display.

FIG. 17 is a view illustrating an example of display of a plurality ofstates.

EXAMPLE EMBODIMENT

Next, an example embodiment of the present invention will be describedwith reference to the drawings.

Each drawing is for describing an example embodiment. However, eachexample embodiment is not limited to the description of the drawings.Similar configurations in the respective drawings are denoted by thesame reference numerals, and repeated description thereof may beomitted.

In the drawings used in the following description, in the description ofeach example embodiment, the description of portions not related to thesolution of the issue of the present invention may be omitted and notillustrated.

Before the description of each example embodiment, first, terms in thefollowing description will be organized.

A “moving object” is a moving object mounting an imaging device thatcaptures an image of a road.

In each example embodiment, the moving object is any moving object. Forexample, the moving object may be a vehicle (four-wheeled vehicle ortwo-wheeled vehicle) or a drone in which an imaging device is installed.Alternatively, the moving object may be a person who holds and moves theimaging device.

The “state of the moving object” is a state or a feature of the movingobject related to the captured image.

For example, when the type (vehicle type) of the vehicle is different,the mounting position, the angle, and the direction with respect to thetraveling direction of the imaging device may be different. Therefore,the vehicle type mounting the imaging device is an example of a featureof the moving object.

Alternatively, the two-wheeled vehicle has a large change in inclinationof the vehicle body as compared with the four-wheeled vehicle.Therefore, the number of wheels is an example of a feature of the movingobject.

Alternatively, when the moving object is moving at a high speed, thereis a case where an object in a captured image is unclear (for example,occurrence of motion blur). In the determination using such an image,the accuracy of determination decreases. That is, the speed of themoving object affects the state of the image used for determination. Asdescribed above, the movement speed of the moving object is an exampleof the state of the moving object.

Alternatively, the acceleration and vibration of the moving object atthe time of imaging affect the captured image. Therefore, theacceleration and the vibration of the moving object are examples of thestate of the moving object.

The “road” is not limited to a road through which vehicles and peoplepass, and includes a structure related to a road. For example, the“road” may include a sign, a white line, a guardrail, a reflectingmirror, a traffic light, and/or a light. Further, in the followingdescription, the “road” is not limited to a road through which vehiclesand persons pass, and may be a passage through which other objects pass.For example, in each example embodiment, the “road” may be a runway, ataxiway, and an apron through which an airplane passes.

The “state of the road” is a state of the road to be determined and astate of the road affecting the determination.

For example, each example embodiment may use a type of deterioration ofa road, such as deterioration of a road surface (cracks (vertical,horizontal, or tortoise-shell), rutting, and/or pot holes, etc.),deterioration of a road surface seal, and fraying of a peripheralportion of the seal, as the state of the road to be determined.

The state of the road to be determined is not limited to thedeterioration of the road surface. For example, each example embodimentmay determine deterioration (for example, blurring of a white line of aroad surface and a road surface sign and/or breakage of a sign) of astructure related to a road as a state of the road to be determined.

The state of the road to be determined is not limited to deteriorationof the road and structures related to the road. For example, a whiteline and a road surface sign attached to a road surface are constructedin such a way as to be reflected at night. Therefore, each exampleembodiment may determine the reflection state of the white line and theroad surface sign as the state of the road to be determined.

Alternatively, each example embodiment may determine the lighting stateof the light installed on the road, the luminance of the light, or theilluminance of the road surface.

Alternatively, the classification of the road (living roads, city roads,prefectural roads, national roads, expressways, and the like) and thenumber of lanes affect the amount of vehicles traveling on the roadsurface, and the like. Therefore, each example embodiment may use theclassification of the road and the number of lanes as the state of theroad that affects the determination.

Alternatively, the type of pavement of the road, the shape of thematerial of the pavement, and the state of the road surface affect thecaptured image. Therefore, each example embodiment may use the type ofpavement of the road, the shape of the material of the pavement, and thestate of the road surface as the state of the road that affects thedetermination.

The type of pavement of the road is asphalt, concrete, stone, brick,gravel, or the like. The type of pavement may include a constructionmethod of a pavement such as a drainage pavement. The material of thepavement that affects the image is grain roughness and/or color.

Alternatively, the manhole has a different appearance in the imagebetween a sunny day and a rainy day. Alternatively, the wetness of theroad surface due to rain changes the state of the captured image.Furthermore, the amount of rainfall affects the size of a puddlecovering the state of the road surface. Thus, the dry and wet conditionson the surface of the road surface affect the image. The surfacetreatment (for example, a straight groove for drainage or a non-slipcircular groove on a slope) of the road surface affects the image. Thatis, the dry and wet conditions of the road surface and the surfacetreatment affect the image. Therefore, each example embodiment may usedry and wet states of the road surface and/or surface treatment as thestate of the road that affects the image.

In the following description, “road deterioration” is used as an exampleof the “state of road” to be determined.

The “external environment” is information obtained by excluding thestate of the moving object and the state of the road from theinformation affecting the determination using the image of the road.

For example, the imaging condition (for example, the time zone and theweather (sunny, cloudy, rainy, after rain, snowy, etc.) of imagecapturing) affects determination using an image. Therefore, the externalenvironment may include an imaging condition (for example, an imagingtime zone and weather).

Alternatively, the surrounding sound at the time of imaging increaseswhen the road is congested. The congestion of the road affects theprogress of deterioration.

Alternatively, when the road surface is deteriorated, vibrationassociated with movement increases in a moving object such as a vehicle.Therefore, when the road surface is deteriorated, in a moving objectsuch as a vehicle, a sound (hereinafter, referred to as “vibrationsound”) generated with vibration during movement is large.

Alternatively, the rainfall amount affects the captured image. The rainsound during rain is proportional to the amount of rain to some extent.

As described above, sound is one of pieces of information effective fordetermining deterioration of a road. Therefore, the external environmentmay include sound (for example, ambient sound, vibration sound, or rainsound).

Alternatively, image quality, size, and the like of an image are one offactors that affect determination. Alternatively, in the case of amoving image, the frame rate affects the image. As such, the externalenvironment may include image specifications, such as image quality,size, and frame rate of the image.

Alternatively, a shadow or the like of a structure installed around aroad may appear in an image of a road surface. Such a shadow affectsdetermination using an image. Therefore, the external environment mayinclude a structure (for example, advertising signboards, signs, treesaround roads, and/or mid- or high-rise buildings (buildings and bridges)installed beside roads) installed around a road.

As described above, the state of the moving object, the state of theroad, and the external environment include many things. Therefore, theuser or the like may appropriately select information to be used as thestate of the moving object, the state of the road, and the externalenvironment in consideration of the determination target.

In the following description, terms related to weather are as follows.However, these terms may be different from weather terms in a strictsense. In a case where the user classifies images to be applied to themodel, the determination may be appropriately made in consideration ofoperational effectiveness and convenience.

The “fine weather” includes “clear weather” in which the cloud amount inthe sky is one or less and “sunny weather” in which the cloud amount is2 or more and 8 or less.

“Cloudy weather” is weather with the cloud amount being nine or more andno precipitation.

“Rainy weather” is weather with rain.

The cloud amount is a ratio of cloud covering the sky, and is an integerof 0 when there is no cloud and 10 when the clouds cover the whole sky.

First Example Embodiment

Next, a first example embodiment will be described with reference to thedrawings.

Description Of Configuration

First, a configuration of a state determination device 10 according tothe first example embodiment will be described with reference to thedrawings.

FIG. 1 is a block diagram illustrating an example of a configuration ofthe state determination device 10 according to the first exampleembodiment.

The state determination device 10 includes a plurality of determinationmodels 110, an output combining unit 120, and a state determination unit130. The number of determination models 110 included in the statedetermination device 10 is not particularly limited as long as it isplural.

The determination model 110 includes a model that determines the stateof the road using the input image of the road. For example, thedetermination model 110 outputs the position of deterioration of theroad and the score thereof using the image of the road.

The determination model 110 is a learned model learned in advance usingartificial intelligence (AI) or machine learning. For example, thedetermination model 110 is a learned model in which the state of theroad is learned using training data including an image obtained byimaging the road and a correct answer label of the state of the roadusing a method such as AI or machine learning.

In the following description, a stage of learning the model (forexample, the determination model 110) using the training data isreferred to as a “learning phase”.

A stage of inputting an image of a road to be determined, determining astate of the road (for example, the location and score of roaddegradation) using a learned model (for example, the determination model110), and outputting a result of the determination is referred to as an“determination phase”.

That is, the determination model 110 is a learned model learned for thedetermination in the learning phase before operating in the actualdetermination phase.

At least some of the determination models 110 may have a structuredifferent from those of other determination models 110. For example, atleast some of the determination model 110 may include a neural networkhaving a structure different from those of other determination models110. However, all the determination models 110 may have the samestructure.

The learning phase of the plurality of determination models 110illustrated in FIG. 1 will be described.

In the learning phase, the determination models 110 are learned for thedetermination of the state of the road using training data includingimages in which at least one of the state of the moving object, thestate of the road, and the external environment is mutually different.

For example, the training data may be different from each other in thestate of the moving object. Alternatively, the training data may bedifferent from each other in the state of the road. Alternatively, thetraining data may be different from each other in the externalenvironment. Alternatively, the training data may be different from eachother in any two of the state of the moving object, the state of theroad, and the external environment. Alternatively, the training data maybe different from each other in all of the state of the moving object,the state of the road, and the external environment.

For three or more pieces of training data, the points in which theimages are different may be different. For example, certain trainingdata may be different from another training data in the state of themoving object, and may be different from different another training datain the state of the road.

The determination model 110 is a learned model learned using mutuallydifferent training data in this manner.

Then, in the learning phase, the determination models 110 are learnedfor the determination of the state of the road using the training dataincluding images in which at least one of the state of the movingobject, the state of the road, and the external environment is mutuallydifferent as described above.

For example, in the learning phase, a certain determination model 110 islearned for the determination of the situation of the road usingtraining data including an images in which the state of the movingobject is different. Then, in the learning phase, another determinationmodel 110 is learned for the determination of the state of the roadusing training data including an image in which the external environmentis different.

For example, before learning is executed, the user of the statedetermination device 10 prepares, as the training data, a plurality ofpieces of training data including images in which at least one of thestate of the moving object, the state of the road, and the externalenvironment is mutually different. For example, each group in which theimages prepared as the training data by the user are divided into groupsin which at least one of the state of the moving object, the state ofthe road, and the external environment is mutually different and labeledmay be used as the training data.

When at least some images included in training data are mutuallydifferent from other images included the training data, some imagesincluded in the training data may be the same as other images.

The amount of images used for the training data may be the same for eachgroup, or the amount of images of the training data of at least somegroups may be different from the amount of images of the training dataof other groups.

For example, the state determination device 10 executes the learningphase of the determination model 110 in such a way that respectivedetermination models 110 learns using mutually different training data.

The state determination device 10 determines that the learning timing ofeach determination model 110 is any timing. For example, the statedetermination device 10 may execute learning of the determination model110 continuously or in parallel. Alternatively, the state determinationdevice 10 may discretely execute learning of at least some of thedetermination models 110.

A device different from the state determination device 10 may execute atleast part of the learning of the determination model 110. For example,the state determination device 10 may acquire the learned determinationmodel 110 from another device (not illustrated) as at least some of thedetermination models 110.

In a case where all the learned determination models 110 are acquiredfrom another device, the state determination device 10 may not includethe function of the learning phase.

Furthermore, the state determination device 10 may execute additionallearning of at least some of the determination models 110.

As described above, the state determination device 10 includes theplurality of determination models 110 each learned using training dataincluding images in which at least one of the state of the movingobject, the state of the road, and the external environment is mutuallydifferent.

In the following description, it is assumed that the determination model110 has been learned except for the description of the learning phase inthe determination model 110.

In the determination phase, the determination model 110 determines thestate of the road using the input image obtained by imaging the road tooutput the determined state of the road, specifically, the deteriorationof the road and the score thereof to the output combining unit 120.

The output combining unit 120 acquires output from the plurality ofdetermination models 110. Then, the output combining unit 120 combinesoutput of the plurality of determination models 110 using apredetermined method.

For example, the output combining unit 120 may store a weight for theoutput of each of the plurality of determination models 110 in advance,and combine output of the determination models 110 using the storedweights.

An example of a combining operation using weights will be described.

The output combining unit 120 stores a weight for each determinationmodel 110 in advance. Then, the determination model 110 outputs thedeterioration position of the road and the score thereof using, forexample, AI for object detection. In this case, the output combiningunit 120 outputs, as a result of a combination, a result obtained bymultiplying the deterioration position output by the determination model110 and the score thereof by a weight for each of the determinationmodels 110 and then combining them.

The output combining unit 120 may store a plurality of weights asweights for each determination model 110 instead of one weight. Forexample, the output combining unit 120 may store a plurality of weights(for example, weights for fine weather, cloudy weather, and rainyweather) related to the state of the image to be determined at the timeof imaging (for example, weather).

An example in which the output combining unit 120 uses a plurality ofweights will be described.

It is assumed that the state determination device 10 includes thefollowing three models as the determination model 110.

(1) The determination model 110 learned using training data of fineweather (hereinafter, “fine weather model”);

(2) The determination model 110 learned using training data of cloudyweather (hereinafter, “cloudy weather model”); and

(3) The determination model 110 learned using training data of rainyweather (hereinafter, “rainy weather model”).

Then, the output combining unit 120 determines the weight using thecloud amount and the precipitation amount.

FIG. 2 is a diagram illustrating an example of a weight.

The output combining unit 120 stores the weight illustrated in FIG. 2 inadvance. Then, the output combining unit 120 combines output of thedetermination model 110 as follows.

The output combining unit 120 acquires the cloud amount and theprecipitation amount of the image to be determined. For example, theoutput combining unit 120 acquires the cloud amount and theprecipitation amount from the transmission source of the image to bedetermined or the like.

A method of acquiring the precipitation amount and the precipitationamount by the output combining unit 120 is any method. For example, theoutput combining unit 120 may acquire an imaging position and an imagingdate and time of an image, and acquire a cloud amount and aprecipitation amount at the acquired position and date and time from acompany or the like that provides weather data. Alternatively, theoutput combining unit 120 may estimate the cloud amount and theprecipitation amount using measurement data of a sensor (for example, anilluminance sensor and a humidity sensor) mounted on the moving object.Alternatively, the output combining unit 120 may estimate the cloudamount and the precipitation amount by applying a predetermined imageanalysis method to the image to be determined.

Then, the output combining unit 120 selects, as the weight for thedetermination model 110, a weight to be used for combination from theweights illustrated in FIG. 2 based on the cloud amount and theprecipitation amount when the image to be determined is captured.

A field of “-” in FIG. 2 indicates that it is not considered in theselection of the weight. For example, in a case where the cloud amountis 1 to 8, the output combining unit 120 determines the weight withoutconsidering the precipitation amount. The output combining unit 120compares the precipitation amount with a predetermined threshold valueto determine the precipitation amount.

For example, when the cloud amount is 6, the output combining unit 120selects “0.7” as the weight for the fine weather model, “0.3” as theweight for the cloudy weather model, and “0.0” as the weight for therainy weather model.

Then, the output combining unit 120 combines output of the fine weathermodel, the cloudy weather model, and the rainy weather model using theweight.

In this manner, the output combining unit 120 combines the results of adetermination by the plurality of determination models 110 using apredetermined method.

In FIG. 2 , the weight “0.0” of the rainy weather model when the cloudamount is 6 means that the output combining unit 120 does not use theoutput of the rainy weather model for combination. In this manner, theoutput combining unit 120 may not combine output of some of thedetermination models 110.

In a case where the total of the weights is not “1”, the outputcombining unit 120 may calculate a value (that is, the weighted averagevalue) obtained by dividing the added value as a result of a combinationby the number of determination models 110.

Alternatively, in a case where the accuracy of a determination by thedetermination model 110 is different, the output combining unit 120 mayuse a weight in consideration of the accuracy of a determination by thedetermination model 110.

In this manner, the output combining unit 120 combines output(determination results) of the plurality of determination models 110using a predetermined method.

However, the method of combining the outputs is not limited to theabove, and may be any method. The user of the state determination device10 may select a method of combining output based on knowledge or thelike.

For example, the output combining unit 120 may output, as the“deterioration position”, a position determined as the deteriorationposition by the determination models 110 whose number is more than thethreshold value as a result of a combination.

Alternatively, the output combining unit 120 may use a combinationmethod learned using AI.

The description returns to the description with reference to FIG. 1 .

The output combining unit 120 outputs the addition result to the statedetermination unit 130.

The state determination unit 130 determines the state of the road in theimage to be determined based on a result of the combination by theoutput combining unit 120.

Each determination model 110 determines the state of the road.Therefore, the result of a combination by the output combining unit 120is the determination of the state of the road. Therefore, the user ofthe state determination device 10 may use the result of a combination bythe output combining unit 120.

However, the state of the road required by the user may be a statedetermined using the result of a combination in addition to the resultof a combination.

For example, in a case where the determination model 110 outputs theposition and type of deterioration, the result of a combination by theoutput combining unit 120 is the position and type of deterioration.Therefore, the user can grasp the position and type of deterioration.

However, for example, when determining necessity of repair, the userneeds the degree of deterioration in addition to the position and typeof deterioration. Therefore, the state determination unit 130 maydetermine the degree of deterioration (hereinafter, referred to as“deterioration”) based on the output position and type of deteriorationas the state of the road. In this manner, the state determination unit130 determines the state of the road using the result of a combinationby the output combining unit 120.

A method of determining the state of the road in the state determinationunit 130 is any method. For example, the state determination unit 130may determine the state of the road using a general image processingmethod.

The state determination unit 130 may include the information (forexample, the location and type of degradation) acquired from the outputcombining unit 120 in the state of the road to be output.

Then, the state determination unit 130 outputs the state of the road,which is the result of the determination, to a predetermined device. Forexample, the state determination unit 130 outputs the determinationresult to a device that displays the state of the road.

The state determination unit 130 may store the state of the road as aresult of the determination in a predetermined storage device.

The state determination unit 130 may output an image used fordetermination in addition to the state of the road.

The state determination device 10 may determine the state of the roadusing a plurality of images (a plurality of still images or movingimages) instead of one image.

Description of Operation

Next, an operation related to the state determination device 10 will bedescribed with reference to the drawings.

(1) Learning Phase

First, the operation in the learning phase of the determination model110 will be described.

FIG. 3 is a flowchart illustrating an example of the operation in thelearning phase of the determination model 110.

It is assumed that the state determination device 10 has previouslyacquired a plurality of pieces of training data in which at least one ofthe state of the moving object mounting the imaging device that acquiresthe image of the road, the state of the road, and the externalenvironment is different.

The state determination device 10 executes the following operation onall the determination models 110.

First, the state determination device 10 selects one determination model110 from the unlearned determination models 110 (step S401).

The state determination device 10 selects one piece of training datafrom training data that is not used for learning (step S402).

Then, the state determination device 10 executes learning of theselected determination model 110 using the selected training data (stepS403). The determination model 110 may use general machine learning aslearning.

The state determination device 10 repeats the above operation untilthere is no unlearned determination model 110.

(2) Determination Phase

Next, the operation in the determination phase in the statedetermination device 10 will be described.

FIG. 4 is a flowchart illustrating an example of the operation in thedetermination phase of the state determination device 10 according tothe first example embodiment.

The determination model 110 has been learned in the determination phase.

The state determination device 10 acquires an image of a road to bedetermined from a predetermined device (step S411). For example, thestate determination device 10 acquires an image of a road captured by animaging device (not illustrated). Alternatively, the state determinationdevice 10 acquires an image of a road stored in a storage device (notillustrated).

Then, each determination model 110 determines the state of the roadusing the image of the road (step S412). For example, each determinationmodel 110 determines road deterioration (for example, the location andtype of degradation). Then, the determination model 110 outputs thedetermination result to the output combining unit 120.

The output combining unit 120 combines output (determination results) ofthe determination model 110 (step S413). Then, the output combining unit120 outputs the combined result to the state determination unit 130.

The state determination unit 130 determines the state of the road (forexample, the deterioration) using the result of the combination by theoutput combining unit 120 (step S414).

Then, the state determination unit 130 outputs the state of the road(for example, the deterioration) to a predetermined device (step S415).The state determination unit 130 may include the result of a combinationby the output combining unit 120 (for example, the location and type ofdegradation) in the state of the road.

Description of Effects

Next, effects of the state determination device 10 according to thefirst example embodiment will be described.

The state determination device 10 according to the first exampleembodiment can obtain effects of improving the accuracy of determinationof a state related to a road.

The reason is as follows.

The state determination device 10 includes the plurality ofdetermination models 110 and the output combining unit 120. Thedetermination models 110 are determination models each learned usingtraining data in which at least one of the state of the moving objectmounting the imaging device that acquires the image of the road, thestate of the road, and the external environment is different. The outputcombining unit 120 combines output from the plurality of determinationmodels for the input image of the road.

As described above, the state determination device 10 combines theresults of a determination by the plurality of determination models 110that has been learned using the training data in which the states of themobile bodies and the like are different. Therefore, the statedetermination device 10 can improve the accuracy of determination on theinput image of the road.

As an example, a case related to weather will be described.

The “fine weather” includes a cloud amount of up to 8. On the otherhand, the “cloudy weather” has a cloud amount of 9 or more. However, theactual cloud amount may be a cloud amount of between 8 and 9. In thedetermination in such a case, it is assumed that the determination inaccordance with the actual cloud amount is made when using the result ofa determination by both models rather than using the result of adetermination by either the fine weather model or the cloudy weathermodel.

As described above, depending on the weather, the accuracy of adetermination of the state of the road can be improved by consideringthe output of the plurality of determination models 110.

Therefore, the output combining unit 120 combines the results of adetermination by the plurality of determination models 110 using apredetermined method (for example, a weight). Therefore, the result of acombination by the output combining unit 120 is a determination withhigher accuracy than that of a determination by one model.

The state determination device 10 further includes the statedetermination unit 130. The state determination unit 130 determines andoutput the state of the road based on a result of the combination by theoutput combining unit 120. As a result, the state determination unit 130determines the state of the road using a result obtained by combiningthe determinations by the plurality of determination models 110, thatis, a more accurate result.

Therefore, the state determination device 10 can improve the accuracy ofa determination of the state related to the road as compared with thedetermination using one determination model.

The determination models 110 may have different accuracies. In such acase, simply using any one of the plurality of determination models 110for determination cannot ensure the accuracy of determination.

For example, a device using any one of the models (hereinafter, referredto as a “related device”) includes a fine weather model, a cloudyweather model, and a rainy weather model. Then, it is assumed that therelated device selects and determines a model in accordance with theweather.

However, for example, it is assumed that the accuracy of the rainyweather model is lower than the accuracy of each of the fine weathermodel and the cloudy weather model.

In this case, the related device uses the rainy weather model for theimage of the rain. Therefore, the related device cannot ensure theaccuracy of determination in rainy weather.

Meanwhile, the state determination device 10 includes the outputcombining unit 120. Then, the output combining unit 120 may combine theoutput in consideration of the accuracy of each of the plurality ofdetermination models 110. Therefore, even in a case where the accuracyof some of the determination models 110 is low, the state determinationdevice 10 can improve the accuracy of determination using the resultobtained by combining the results of a determination by otherdetermination models 110.

Hardware Configuration

Next, a hardware configuration of the state determination device 10 willbe described.

Each component of the state determination device 10 may be configured bya hardware circuit.

Alternatively, in the state determination device 10, each component maybe configured using a plurality of devices connected via a network. Forexample, the state determination device 10 may be configured using cloudcomputing.

Alternatively, in the state determination device 10, the plurality ofcomponents may be configured by one piece of hardware.

Alternatively, the state determination device 10 may be achieved as acomputer device including a central processing unit (CPU), a read onlymemory (ROM), and a random access memory (RAM). In addition to the aboveconfiguration, the state determination device 10 may be implemented as acomputer device including a network interface circuit (NIC).

FIG. 5 is a block diagram illustrating an example of a hardwareconfiguration of the state determination device 10.

The state determination device 10 includes a CPU 610, a ROM 620, a RAM630, a storage device 640, and an NIC 650, and thus is implemented as acomputer device.

The CPU 610 reads a program from the ROM 620 and/or the storage device640. Then, the CPU 610 controls the RAM 630, the storage device 640, andthe NIC 650 based on the read program. Then, the computer including theCPU 610 controls these configurations to achieve the functions as thedetermination model 110, the output combining unit 120, and the statedetermination unit 130 illustrated in FIG. 1 .

When achieving each function, the CPU 610 may use the RAM 630 or thestorage device 640 as a temporary storage medium of the program.

The CPU 610 may read the program included in a recording medium 690storing the program in a computer readable manner using a recordingmedium reading device (not illustrated). Alternatively, the CPU 610 mayreceive a program from an external device (not illustrated) via the NIC650, store the program in the RAM 630 or the storage device 640, andoperate based on the stored program.

The ROM 620 stores programs executed by the CPU 610 and fixed data. TheROM 620 is, for example, a programmable ROM (P-ROM) or a flash ROM.

The RAM 630 temporarily stores programs and data executed by the CPU610. The RAM 630 is, for example, a dynamic-RAM (D-RAM).

The storage device 640 stores data and programs to be stored for a longperiod of time by the state determination device 10. The storage device640 may operate as a temporary storage device of the CPU 610. Thestorage device 640 is, for example, a hard disk device, amagneto-optical disk device, a solid state drive (SSD), or a disk arraydevice.

The ROM 620 and the storage device 640 are non-transitory recordingmedia. On the other hand, the RAM 630 is a transitory recording medium.The CPU 610 can operate based on a program stored in the ROM 620, thestorage device 640, or the RAM 630. That is, the CPU 610 can operateusing a non-transitory recording medium or a transitory recordingmedium.

The NIC 650 relays exchange of data with an external device (forexample, the transmission source of the image for determination and theoutput destination device of the state of the road) not illustrated viaa network. The NIC 650 is, for example, a local area network (LAN) card.Furthermore, the NIC 650 is not limited to use wired communication, butmay use wireless communication.

The state determination device 10 of FIG. 5 configured as describedabove can obtain the effects same as those of the state determinationdevice 10 of FIG. 1 .

This is because the CPU 610 of the state determination device 10 in FIG.5 can implement each function of the state determination device 10 inFIG. 1 based on the program.

System Configuration

FIG. 6 is a block diagram illustrating an example of a configuration ofa state determination system 50 including the state determination device10 according to the first example embodiment.

The state determination system 50 includes the state determinationdevice 10, an imaging device 20 and/or an image storage device 25, and adisplay device 30.

The imaging device 20 is mounted on a moving object, acquires trainingdata and/or an image to be determined, and outputs the training dataand/or the image to the state determination device 10. Alternatively,the imaging device 20 may acquire training data and/or an image to bedetermined, and store the training data and/or the image to bedetermined in the image storage device 25. The imaging device 20 is, forexample, a camera or a drive recorder.

The number of imaging devices 20 may be one or plural.

FIG. 15 is a diagram illustrating an example of a case where a pluralityof drive recorders 820 is used as the imaging device. FIG. 15illustrates a vehicle 810 as an example of the moving object.

A network 850 is a communication path connected to the statedetermination device 10.

A wireless communication path 830 connects the drive recorder 820 and aradio base station 840.

The radio base station 840 relays the network 850 to which the statedetermination device 10 is connected and the wireless communication path830.

The vehicle 810 mounts the drive recorder 820 and travels on a road.

The drive recorder 820 is mounted on the vehicle 810, and captures animage of a road on which the mounted vehicle 810 travels. FIG. 15illustrates the drive recorder 820 adjacent to the outside of thevehicle 810 for easy understanding.

Then, the drive recorder 820 transmits the captured image to the statedetermination device 10 via the wireless communication path 830, theradio base station 840, and the network 850. The wireless communicationpath 830, the radio base station 840, and the network 850 are an exampleof a communication path between the drive recorder 820 and the statedetermination device 10. The drive recorder 820 may be connected to thestate determination device 10 by a device or a path different from thatin FIG. 15 .

The vehicle 810, the drive recorder 820, the wireless communication path830, the radio base station 840, and the network 850 are notparticularly limited. The vehicle 810, the drive recorder 820, thewireless communication path 830, the radio base station 840, and thenetwork 850 may be generally sold products. Therefore, a detaileddescription thereof will be omitted.

The state determination device 10 may be mounted on a moving object. Inthis case, the state determination device 10 may acquire thedetermination image from the imaging device 20 directly or via thecommunication path in the moving object.

The description returns to the description with reference to FIG. 6 .

The image storage device 25 stores training data and/or an image to bedetermined. Then, the image storage device 25 outputs the training dataand/or the image to be determined to the state determination device 10.The image storage device 25 is, for example, a hard disk device, amagneto-optical disk device, an SSD, or a disk array device.

The image storage device 25 receives training data and/or an image to bedetermined from the imaging device 20. However, the image storage device25 may receive training data and/or an image to be determined from adevice (not illustrated) different from the imaging device 20.

Whether the state determination device 10 acquires the training data andthe image to be determined from the imaging device 20 or the imagestorage device 25 may be appropriately determined by the user. Forexample, the state determination device 10 may acquire training datafrom the image storage device 25 and acquire the image to be determinedfrom the imaging device 20.

The state determination device 10 operates as described above, andexecutes the learning phase using the training data.

Further, the state determination device 10 operates as described above,determines the state of the road using the image to be determined, andoutputs the determined state of the road as the determination phase.

The display device 30 displays the result (state of road) of thedetermination output by the state determination device 10. The displaydevice 30 is, for example, a liquid crystal display, an organicelectroluminescence display, or electronic paper.

The display on the display device 30 is any display. The user mayappropriately display the state of the road on the display device 30 asnecessary.

FIG. 16 is a diagram illustrating an example of display.

FIG. 16 illustrates a deterioration position using a rectangular framein the image of the road to be determined. Further, FIG. 16 highlights(hatched) the position where the state determination device 10 hasdetermined as “deterioration is large”.

The display device 30 may collectively display not one but a pluralityof states of the road as the display of the state of the road determinedby the state determination device 10.

FIG. 17 is a diagram illustrating an example of display of a pluralityof states.

FIG. 17 illustrates a portion determined to be deteriorated on a road ina predetermined region using an arrow. Further, FIG. 17 highlights (inblack) the “large deterioration”. The direction of the arrow in FIG. 17is the traveling direction of the vehicle on the road.

Second Example Embodiment

Next, a second example embodiment will be described in detail withreference to the drawings.

Description of Configuration

FIG. 7 is a block diagram illustrating an example of a configuration ofa state determination device 11 according to the second exampleembodiment.

The state determination device 11 includes a plurality of determinationmodels 110, an output combining unit 121, a state determination unit130, and a weight determination unit 140. The determination model 110and the state determination unit 130 are similar to the determinationmodel 110 and the state determination unit 130 of the first exampleembodiment. Therefore, a detailed description thereof will be omitted.

The state determination device 11 may be configured using a computerillustrated in FIG. 5 . Alternatively, the state determination system 50may include the state determination device 11 instead of the statedetermination device 10.

The output combining unit 121 combines the results of a determination bythe determination model 110 using the weight determined by the weightdetermination unit 140. For example, when the state determined by thedetermination model 110 is the deterioration position and the score, theoutput combining unit 121 combines the deterioration position and thescore output by the determination model 110 using the weight determinedby the weight determination unit 140.

The output combining unit 121 operates as in the output combining unit120 of the first example embodiment except that the weight determined bythe weight determination unit 140 is used.

The weight determination unit 140 determines a weight to be used whenthe output combining unit 121 combines the determination models 110using the image to be determined.

Therefore, the weight determination unit 140 stores a mechanism (forexample, an equation for calculating a weight or a determinant) fordetermining the weight using the image in advance. In the followingdescription, a mechanism for determining a weight is referred to as a“weight determination formula”. However, the weight determinationformula is not limited to a simple scalar formula. For example, theweight determination formula may be an equation, a vector formula, adeterminant, a function, or a combination thereof. The weightdetermination formula may include a function including a conditionalstatement.

An operation of the weight determination unit 140 will be describedusing a simple example.

For example, the following Equation (1) is a weight determinationformula representing combination by the output combining unit 121.

Y=a _(i)(p)X _(i)(p)+a ₂(p)X ₂(p)+ . . . +a _(n)(p)X _(n)(p)   (1)

In the above Equation (1), p is an image to be determined (Specifically,for example, the value of the pixel of the image). n is the number ofdetermination models 110. i is a variable indicating the determinationmodel 110 (i=1, . . . , n). Y is the result of a combination. a_(i)(p)(i=1, . . . , n) is a function that determines a weight for eachdetermination model 110 in the case of the image p. X_(i)(p) (i=1, 2, .. . , n) is output of the determination model 110.

A data format such as a variable in Equation (1) may be appropriatelyselected from a scalar, a vector, a matrix, a function, or a combinationthereof. The coefficient of the function a_(i)(p) is an example of theparameter of the weight determination formula.

In this case, the weight determination unit 140 determines the weight byapplying the image p to the function a_(i)(p). The determination model110 calculates “X_(i)(p)”, which is a result of the determination, usingthe image p. Then, the output combining unit 120 calculates “Y”, whichis a result of a combination, using the weight (a_(i)(p)) determined bythe weight determination unit 140 and the result of the determination(X_(i)(p)) output by the determination model 110.

The weight determination formula is stored in advance in the statedetermination device 11.

The configuration for storing the weight determination formula is anyconfiguration. The state determination device 11 may store the weightdetermination formula in a storage unit (not illustrated). In this case,the weight determination unit 140 acquires the weight determinationformula stored in the storage unit as necessary, applies the image tothe weight determination formula to determine the weight, and providesthe determined weight to the output combining unit 121. Alternatively,the weight determination unit 140 may store the weight determinationformula. Alternatively, when determining the weight, the weightdetermination unit 140 may acquire the weight determination formula froma device that stores a weight determination formula (not illustrated).

The weight determination unit 140 may use a weight determination formulalearned in advance using a predetermined method such as AI. For example,the weight determination unit 140 may use a weight determination formulathat has learned a parameter (a coefficient of a_(i)(p)) usingpredetermined training data for learning the weights.

Description of Operation

Next, an operation in the determination phase in the state determinationdevice 11 according to the second example embodiment will be describedwith reference to the drawings.

FIG. 8 is a flowchart illustrating an example of the operation in thedetermination phase of the state determination device 11 according tothe second example embodiment.

The determination model 110 has been learned.

The state determination device 11 acquires an image of a road to bedetermined from a predetermined device (step S411). For example, thestate determination device 11 may acquire an image of a road captured bythe imaging device 20. Alternatively, the state determination device 11may acquire an image of a road stored in the image storage device 25.

Then, each determination model 110 determines the state of the roadusing the image of the road (step S412). For example, each determinationmodel 110 determines road deterioration (for example, the location andtype of degradation). Then, the determination model 110 outputs thedetermination result to the output combining unit 120.

The weight determination unit 140 determines the weight using the image(step S416).

The output combining unit 121 combines output (determination results) ofthe determination model 110 using the determined weights (step S417).Then, the output combining unit 121 outputs the combined result to thestate determination unit 130.

The state determination unit 130 determines the state of the road usingthe result of the combination by the output combining unit 121 (stepS414).

Then, the state determination unit 130 outputs the state of the road(for example, the deterioration) to a predetermined device (step S415).The state determination unit 130 may include the result of a combinationby the output combining unit 121 (for example, the location and type ofdegradation) in the state of the road.

Description of Effects

Next, effects of the state determination device 11 according to thesecond example embodiment will be described.

The state determination device 11 according to the second exampleembodiment can obtain the effects same as those of the statedetermination device 10 according to the first example embodiment.

The reason is as follows.

This is because the output combining unit 121 operates using the weightdetermined by the weight determination unit 140 as in the outputcombining unit 120.

Furthermore, the state determination device 11 according to the secondexample embodiment can obtain an effect of further improving theaccuracy of determination.

The reason is as follows.

The weight determination unit 140 determines the weight to be used bythe output combining unit 121 based on the image to be determined. Thatis, the weight determination unit 140 determines the weight inaccordance with the image to be determined.

Then, the output combining unit 121 combines output of the determinationmodels 110 using the weight related to the image to be determined. Thatis, the output combining unit 121 can execute the more appropriatecombination.

As a result, the state determination device 11 can further improve theaccuracy of a determination of the state of the road.

Third Example Embodiment

Next, a third example embodiment will be described with reference to thedrawings.

Description of Configuration

FIG. 9 is a block diagram illustrating an example of a configuration ofa state determination device 12 according to the third exampleembodiment.

The state determination device 12 includes a plurality of determinationmodels 110, an output combining unit 121, a state determination unit130, a weight determination unit 141, a loss calculation unit 150, and aparameter correction unit 160.

The state determination device 12 may be configured using a computerillustrated in FIG. 5 . Alternatively, the state determination system 50may include the state determination device 12 instead of the statedetermination devices 10 and 11.

The determination model 110 and the state determination unit 130 aresimilar to the determination model 110 and the state determination unit130 of the first example embodiment and the second example embodiment.The output combining unit 121 is similar to the output combining unit121 of the second example embodiment.

Therefore, detailed description of configurations and operations similarto those of the first example embodiment and the second exampleembodiment will be omitted, and configurations and operations specificto the third example embodiment will be described.

The loss calculation unit 150 and the parameter correction unit 160operate in the weight learning phase in the weight determination unit141. The weight determination unit 141 operates as in the weightdetermination unit 140 except that it learns the weight in cooperationwith the loss calculation unit 150 and the parameter correction unit160.

Therefore, a configuration related to the learning phase in the weightdetermination unit 141 will be described.

The loss calculation unit 150 calculates a difference (loss) betweendata (for example, a deterioration position and its score in each image,hereinafter referred to as “correct answer label”) indicating a correctanswer for the training data for learning the weights and a result of acombination by the output combining unit 121. In the followingdescription, data indicating a correct answer is referred to as a“correct answer label”. In the following description, the losscalculated by the loss calculation unit 150 is referred to as a“combination loss” or a “first loss”.

A configuration for storing the correct answer label is anyconfiguration. The loss calculation unit 150 may store a correct answerlabel related to the training data for learning the weights in advance.Alternatively, the state determination device 12 may store the correctanswer label in a storage unit (not illustrated). Alternatively, in thecalculation of the loss, the loss calculation unit 150 may acquire thecorrect answer label from a device or a configuration that storestraining data for learning the weights.

The parameter correction unit 160 corrects the parameter (for example,the coefficient of the function a_(i)(p) of Equation (1)) of the weightdetermination formula used by the weight determination unit 140, basedon the loss calculated by the loss calculation unit 150.

Description of Operation

Next, an operation of the state determination device 12 according to thethird example embodiment will be described with reference to thedrawings.

FIG. 10 is a flowchart illustrating an example of the operation in thelearning phase of the weight determination unit 141 according to thethird example embodiment.

The weight determination unit 141 stores a predetermined weightdetermination formula (for example, the weight determination formula inwhich the coefficient of the function a_(i)(p) has a predeterminedvalue) as an initial value of the weight before the operation in thelearning phase.

The state determination device 12 acquires an image as training data forlearning the weights (step S421).

The state determination device 12 selects one image from the trainingdata for learning the weights. Then, each of the plurality ofdetermination models 110 determines the state of the road using theselected image (step S422). Then, the determination model 110 outputsthe determination result to the output combining unit 121.

The weight determination unit 141 applies the selected image to theweight determination formula to determine the weight (step S423).

The output combining unit 121 combines output of the determinationmodels 110 using the weights determined by the weight determination unit141 (step S424).

The loss calculation unit 150 calculates a loss (combination loss)between the correct answer label and the result of a combination by theoutput combining unit 121 (step S425).

The parameter correction unit 160 corrects the parameter (for example,the coefficient of the function a_(i)(p)) of the weight determinationformula based on the loss (combination loss) calculated by the losscalculation unit 150 (step S426).

The state determination device 12 determines whether learning isfinished (step S427). Specifically, the state determination device 12determines whether a predetermined end condition (for example, the valueof loss is smaller than a threshold value, perform execution aprescribed number of times, or termination of training data for learningthe weights) is satisfied.

When the end condition is not satisfied (No in step S427), the statedetermination device 12 returns the process to step S422.

When the end condition is satisfied (Yes in step S427), the statedetermination device 12 ends the learning phase.

In this manner, the state determination device 12 learns the weight (forexample, the parameter of the weight determination formula) determinedby the weight determination unit 141 using the training data forlearning the weights.

In the determination phase, the state determination device 12 operatesas in the state determination device 11 of the second exampleembodiment. That is, in the determination phase, the state determinationdevice 12 determines the state of the road using the determination model110, the output combining unit 121, the state determination unit 130,and the weight determination unit 141.

The operation in the determination phase of the state determinationdevice 12 will be described with reference to FIG. 8 .

The state determination device 12 acquires an image of a road to bedetermined from a predetermined device (step S411).

The determination model 110 outputs the state of the road using theimage of the road (step S412).

The weight determination unit 141 determines the weight by applying theimage of the road to the learned weight determination formula (stepS416).

The output combining unit 121 combines output of the determinationmodels 110 using the determined weights (step S417).

The state determination unit 130 determines the state of the road usingthe result of a combination (step S414).

The state determination unit 130 outputs the determination result (stateof the road) to a predetermined device (step S415).

Description of Effects

Next, effects of the state determination device 12 according to thethird example embodiment will be described.

The state determination device 12 according to the third exampleembodiment can obtain an effect of further improving determinationaccuracy in addition to the effects of the first example embodiment andthe second example embodiment.

The reason is as follows.

In the learning phase, the weight determination unit 141 learns theweight using the training data for learning the weights. Then, in thedetermination phase, the weight determination unit 141 determines theweight using the learned weight determination formula.

The output combining unit 121 combines output of the determinationmodels 110 using the determined weights. That is, the output combiningunit 121 combines the output using the weights learned using thetraining data for learning the weights. Therefore, the output combiningunit 121 can execute more appropriate combination.

As a result, the state determination device 12 can further improve theaccuracy of a determination of the state of the road.

Fourth Example Embodiment

Next, a fourth example embodiment will be described with reference tothe drawings.

Description of Configuration

FIG. 11 is a block diagram illustrating an example of a configuration ofa state determination device 13 according to the fourth exampleembodiment.

The state determination device 13 includes a plurality of determinationmodels 110, an output combining unit 121, a state determination unit130, a weight determination unit 142, a loss calculation unit 150, aparameter correction unit 161, and an external environment losscalculation unit 170.

The state determination device 13 may be configured using a computerillustrated in FIG. 5 .

The determination model 110 and the state determination unit 130 aresimilar to the determination model 110 and the state determination unit130 of the first to third example embodiments. The output combining unit121 is similar to the output combining unit 121 of the second exampleembodiment and the third example embodiment. The loss calculation unit150 is similar to the loss calculation unit 150 of the third exampleembodiment.

Therefore, detailed description of configurations and operations similarto those of the first to third example embodiments will be omitted, andconfigurations and operations specific to the fourth example embodimentwill be described.

The loss calculation unit 150, the parameter correction unit 161, andthe external environment loss calculation unit 170 operate in the weightlearning phase in the weight determination unit 142. The weightdetermination unit 142 operates as in the weight determination unit 141except for the operation of learning the weight in the learning phase.

Therefore, a configuration related to the learning phase in the weightdetermination unit 142 will be described.

As in the weight determination unit 141, the weight determination unit142 determines a weight using an image of training data for learning theweights. Furthermore, the weight determination unit 142 estimates theexternal environment using the image to output the estimated externalenvironment to the external environment loss calculation unit 170.

The weight determination unit 142 desirably shares at least part of theprocessing in the calculation of the weight and the estimation of theexternal environment.

For example, in a case where the weight determination unit 142 uses adeep neural network, the weight determination unit 142 may make a lowerlayer of the deep neural network common and make an upper layerspecialized in calculation of a weight and estimation of an externalenvironment. In this case, the weight determination unit 142 shares someof the parameters for determining the weight and some of the parametersfor estimating the external environment common. In this case, thelearning by the weight determination unit 142 is so-called multi-tasklearning.

Then, since the weight determination unit 142 uses a deep neural networkin which the calculation of the weight and the estimation of theexternal environment are partially shared, the weight for thedetermination model 110 learned using a similar external environmenttends to be a similar weight. As a result, the state determinationdevice 13 can obtain an effect of stabilizing learning by the weightdetermination unit 142.

However, the weight determination unit 142 may use different processingin the calculation of the weight and the estimation of the externalenvironment.

The external environment loss calculation unit 170 acquires an externalenvironment related to an image of training data for learning theweights. An acquisition source of the external environment is anyacquisition source. The external environment loss calculation unit 170may acquire an external environment related to an image of training datafor learning the weights from a provider of training data for learningthe weights.

Alternatively, the external environment loss calculation unit 170 mayacquire the external environment from a device (not illustrated) withreference to training data for learning the weights. For example, theexternal environment loss calculation unit 170 may acquire weather datasuch as weather from a company or the like that provides the weatherdata with reference to the imaging date and time and place of the imageof the training data for learning the weights.

Then, the external environment loss calculation unit 170 calculates adifference (loss) between the acquired external environment and theexternal environment estimated by the weight determination unit 142. Inthe following description, the loss calculated by the externalenvironment loss calculation unit 170 is referred to as an “externalenvironment loss” or a “second loss”.

Then, the external environment loss calculation unit 170 outputs thecalculated loss (external environmental loss) to the parametercorrection unit 161.

The parameter correction unit 161 corrects the parameter of the weightdetermination formula used by the weight determination unit 142 tocalculate the weight, based on a loss (combination loss) calculated bythe loss calculation unit 150 and a loss (external environment loss)calculated by the external environment loss calculation unit 170.

Description of Operation

Next, an operation of the state determination device 13 according to thefourth example embodiment will be described with reference to thedrawings.

FIG. 12 is a flowchart illustrating an example of the operation in thelearning phase of the weight determination unit 142 according to thefourth example embodiment.

The weight determination unit 142 stores a predetermined weightdetermination formula as an initial value of the weight before theoperation in the learning phase.

Steps S421 to S425 and S427 are operations similar to those in FIG. 10 .

The state determination device 13 acquires an image as training data forlearning the weights (step S421).

The state determination device 13 selects one image from the trainingdata for learning the weights. Then, each of the plurality ofdetermination models 110 determines the state of the road using theselected image (step S422). Then, the determination model 110 outputsthe determination result to the output combining unit 121.

The weight determination unit 142 applies the selected image to theweight determination formula to determine the weight (step S423).

The output combining unit 121 combines output of the determinationmodels 110 using the weights determined by the weight determination unit141 (step S424).

The loss calculation unit 150 calculates a loss (combination loss)between the correct answer label and the result of a combination by theoutput combining unit 121 (step S425).

Furthermore, the weight determination unit 142 estimates the environmentusing the image (step S431).

The external environment loss calculation unit 170 calculates theexternal environment loss (step S432).

The parameter correction unit 161 corrects the parameter of the weightdetermination formula used by the weight determination unit 142 tocalculate the weight, based on the loss (combination loss) calculated bythe loss calculation unit 150 and the loss (external environment loss)calculated by the external environment loss calculation unit 170 (stepS433).

The state determination device 13 determines whether learning isfinished (step S427). Specifically, the state determination device 13determines whether a predetermined end condition (for example, the valueof any loss or both losses is smaller than a threshold value, performexecution a prescribed number of times, or termination of training datafor learning the weights) is satisfied.

When the end condition is not satisfied (No in step S427), the statedetermination device 13 returns the process to step S422.

When the end condition is satisfied (Yes in step S427), the statedetermination device 13 ends the learning phase.

The state determination device 13 may replace the execution order of theoperations from steps S422 to S425 and the operations from S431 to S432.Alternatively, the state determination device 13 may execute at leastsome of the operations from steps S422 to S425 and the operations fromS431 to S432 in parallel.

In this manner, the state determination device 13 learns the weightdetermined by the weight determination unit 142 using the training datafor learning the weights and the external environment.

In the determination phase, the state determination device 13 operatesas in the state determination device 11 of the second example embodimentand the state determination device 12 of the third example embodiment.That is, in the determination phase, the state determination device 13determines the state of the road using the determination model 110, theoutput combining unit 121, the state determination unit 130, and theweight determination unit 142.

The operation in the determination phase of the state determinationdevice 13 will be described with reference to FIG. 8 .

The state determination device 13 acquires an image of a road to bedetermined from a predetermined device (step S411).

The determination model 110 outputs the state of the road using theimage of the road (step S412).

The weight determination unit 142 determines the weight by applying theimage of the road to the learned weight determination formula (stepS412).

The output combining unit 121 combines output of the determinationmodels 110 using the determined weights (step S417).

The state determination unit 130 determines the state of the road usingthe result of a combination (discarding step S414).

The state determination unit 130 outputs the determination result (stateof the road) to a predetermined device (step S415).

Description of Effects

Next, effects of the state determination device 13 according to thefourth example embodiment will be described.

The state determination device 13 according to the fourth exampleembodiment can obtain an effect of further improving the accuracy ofdetermination in addition to the effects of the first to third exampleembodiments.

The reason is as follows.

In the learning phase, the weight determination unit 142 learns theparameter of the weight determination formula using the externalenvironment in addition to the training data for learning the weights.Then, in the determination phase, the weight determination unit 142determines the weight using the learned weight determination formula.

The output combining unit 121 combines output of the determinationmodels 110 using the determined weights. That is, the output combiningunit 121 combines the outputs using the weights learned using theexternal environment in addition to the training data for learning theweights. Therefore, the output combining unit 121 can execute moreappropriate combination.

As a result, the state determination device 13 can further improve theaccuracy of a determination of the state of the road.

System

FIG. 13 is a block diagram illustrating an example of a configuration ofa state determination system 51 including the state determination device13 according to the fourth example embodiment.

The state determination system 51 includes the state determinationdevice 13, an imaging device 20 and/or an image storage device 25, adisplay device 30, and an information providing device 40.

The imaging device 20, the image storage device 25, and the displaydevice 30 are similar to those of the first example embodiment.Therefore, a detailed description thereof will be omitted.

The information providing device 40 outputs the external environment tothe state determination device 13. The imaging device may output atleast part of the external environment to the state determination device13.

Whether the state determination device 13 acquires the externalenvironment from the information providing device 40 or the imagingdevice 20 may be appropriately determined by a user or the like.

Then, the state determination device 13 operates as described above tooutput the state of the road to the display device 30 as a result of thedetermination.

Fifth Example Embodiment

A fifth example embodiment will be described with reference to thedrawings.

Description of Configuration

FIG. 14 is a diagram illustrating an example of a configuration of astate determination device 14 according to the fifth example embodiment.

The state determination device 14 includes a plurality of determinationmodels 110 and an output combining unit 120. The determination model 110is similar to the determination model 110 of the first to fourth exampleembodiments. The output combining unit 120 is similar to the outputcombining unit 120 of the first example embodiment.

Each configuration of the state determination device 14 operates as inthe related configuration in the state determination device 10 of thefirst example embodiment or the like.

The state determination device 14 may be configured using a computerillustrated in FIG. 5 . Alternatively, the state determination system 50may include the state determination device 14 instead of the statedetermination devices 10 to 12.

Description of Effects

The state determination device 14 has an effect of improving theaccuracy of a determination of a state related to a road.

The reason is as follows.

The state determination device 14 includes the plurality ofdetermination models 110 and the output combining unit 120. Thedetermination models 110 are determination models each learned usingtraining data in which at least one of the state of the moving objectmounting the imaging device that acquires the image of the road, thestate of the road, and the external environment is different. The outputcombining unit 120 combines output from the plurality of determinationmodels for the input image of the road.

In this manner, each configuration of the state determination device 14operates as in the related configuration in the state determinationdevice 10 of the first example embodiment and the like. That is, theoutput combining unit 120 of the state determination device 14 combinesthe results of a determination by the plurality of determination models110 using a predetermined method. Therefore, the state determinationdevice 14 can output the state of the road with higher accuracy ascompared with the case of using one model.

The user may use a result of a combination by the output combining unit120 in the state determination device 14 as the state of the road.Alternatively, the user may determine the state of the road by providingoutput of the state determination device 14 to a device (notillustrated) that determines the state of the road in detail.

The state determination device 14 according to the fifth exampleembodiment has the minimum configuration in each example embodiment.

Some or all of the above example embodiments may be described as thefollowing Supplementary Notes, but are not limited to the following.

Supplementary Note 1

A state determination device includes:

-   -   a plurality of determination models each learned using training        data in which at least one of a state of a moving object        mounting an imaging device that acquires an image of a road, a        state of the road, and an external environment is different, and    -   an output combining means configured to combine output from the        plurality of determination models for an input image of the        road.

Supplementary Note 2

The state determination device according to Supplementary Note 1,further includes:

-   -   a state determination means configured to determine and output a        state of the road based on a result of the combination by the        output combining means.

Supplementary Note 3

The state determination device according to Supplementary Note 1 or 2,wherein

-   -   the plurality of determination models is respectively learned        using a plurality of pieces of training data including images of        the road in which at least one of a state of the moving object,        a state of the road, and the external environment is mutually        different.

Supplementary Note 4

The state determination device according to any one of SupplementaryNotes 1 to 3, further includes:

-   -   a weight determination means configured to determine weights to        be used for combining output of the plurality of determination        models in accordance with an input image of the road, wherein    -   the output combining means combines output of the plurality of        determination models using the determined weights.

Supplementary Note 5

The state determination device according to Supplementary Note 4,wherein

-   -   the weight determination means learns a parameter for        determining the weights using training data for learning the        weights.

Supplementary Note 6

The state determination device according to Supplementary Note 5,wherein

-   -   the weight determination means further learns the parameter for        determining the weights in accordance with the external        environment.

Supplementary Note 7

A state determination system including:

-   -   the state determination device according to any one of        Supplementary Notes 1 to 5;    -   an imaging device that acquires an image of a road to be        determined to output the image to the state determination        device, and/or a storage device that stores the image of the        road to be determined; and    -   a display device that displays a state of a road output by the        state determination device.

Supplementary Note 8

A state determination system includes:

-   -   the state determination device according to Supplementary Note        6;    -   an imaging device that captures an image of a road to be        determined to output the image to the state determination        device, and/or a storage device that stores the image of the        road to be determined;    -   an information providing device that outputs an external        environment to the state determination device; and    -   a display device that displays a state of a road output by the        state determination device.

Supplementary Note 9

A state determination method includes:

-   -   by a state determination device including a plurality of        determination models each learned using training data in which        at least one of a state of a moving object mounting an imaging        device that acquires an image of a road, a state of the road,        and an external environment is different,    -   combining output from the plurality of determination models for        an input image of a road.

Supplementary Note 10

The state determination method according to Supplementary Note 9,further includes:

-   -   determining a state of the road based on a result of the        combination.

Supplementary Note 11

The state determination method according to Supplementary Note 9 or 10,further includes:

-   -   learning each of the plurality of determination models using a        plurality of pieces of training data including images of the        road in which at least one of a state of the moving object, a        state of the road, and the external environment is mutually        different.

Supplementary Note 12

The state determination method according to any one of SupplementaryNotes 9 to 11, further includes:

-   -   determining weights to be used for combining output of the        plurality of determination models in accordance with an input        image of the road; and    -   combining output of the plurality of determination models using        the determined weights.

Supplementary Note 13

The state determination method according to Supplementary Note 12,further includes:

-   -   learning a parameter for determining the weights using training        data for learning the weights.

Supplementary Note 14

The state determination method according to Supplementary Note 13,further includes:

-   -   learning the parameter for determining the weights in accordance        with an external environment.

Supplementary Note 15

A state determination method includes:

-   -   by a state determination device, executing the state        determination method according to any one of Supplementary Notes        9 to 13;    -   by an imaging device, acquiring an image to be determined to        output the image to the state determination device, and/or, by        the storage device, storing the image to be determined; and    -   by a display device, displaying a state of a road output by the        state determination device.

Supplementary Note 16

A state determination method includes:

-   -   by a state determination device, executing the state        determination method according to Supplementary Note 14;    -   by an imaging device, acquiring an image of a road to be        determined to output the image to the state determination        device, and/or a storage device storing the image to be        determined;    -   by an information providing device, outputting an external        environment to the state determination device; and    -   by a display device, displaying a state of a road output by the        state determination device.

Supplementary Note 17

A recording medium that records a program for causing a computer, thecomputer including a plurality of determination models each learnedusing training data in which at least one of a state of a moving objectmounting an imaging device that acquires an image of a road, a state ofthe road, and an external environment is different, to execute:

-   -   a processing of combining output from the plurality of        determination models for an input image of the road.

Supplementary Note 18

The recording medium according to Supplementary Note 17, wherein themedium records the program for causing the computer to execute:

-   -   a processing of determining a state of the road based on a        result of the combination.

Supplementary Note 19

The recording medium according to Supplementary Note 17 or 18, whereinthe medium records the program for causing the computer to execute:

-   -   a processing of learning each of the plurality of determination        models using a plurality of pieces of training data including        images of the road in which at least one of a state of the        moving object, a state of the road, and the external environment        is mutually different.

Supplementary Note 20

The recording medium according to any one of Supplementary Notes 17 to19, wherein the medium records the program for causing the computer toexecute:

-   -   a processing of determining weights to be used for combining        output of the plurality of determination models in accordance        with an input image of the road; and    -   a processing of combining output of the plurality of        determination models using the determined weights.

Supplementary Note 21

The recording medium according to Supplementary Note 20, wherein themedium records the program for causing the computer to execute:

-   -   a processing of learning a parameter for determining the weights        using training data for learning the weights.

Supplementary Note 22

The recording medium according to Supplementary Note 21, wherein themedium records the program for causing the computer to execute:

-   -   a processing of learning the parameter for determining the        weights in accordance with the external environment.

Although the present invention is described above with reference to theexample embodiments, the present invention is not limited to the aboveexample embodiments. It will be understood by those of ordinary skill inthe art that various changes in form and details may be made thereinwithout departing from the spirit and scope of the present invention asdefined by the claims.

REFERENCE SIGNS LIST

-   -   10 state determination device    -   11 state determination device    -   12 state determination device    -   13 state determination device    -   20 imaging device    -   25 image storage device    -   30 display device    -   40 information providing device    -   50 state determination system    -   51 state determination system    -   110 determination model    -   120 output combining unit    -   121 output combining unit    -   130 state determination unit    -   140 weight determination unit    -   141 weight determination unit    -   142 weight determination unit    -   150 loss calculation unit    -   160 parameter correction unit    -   161 parameter correction unit    -   170 external environment loss calculation unit    -   610 CPU    -   620 ROM    -   630 RAM    -   640 storage device    -   650 NIC    -   690 recording medium    -   810 vehicle    -   820 drive recorder    -   830 wireless communication path    -   840 radio base station    -   850 network

What is claimed is:
 1. A state determination device comprising: amemory; and at least one processor coupled to the memory, the processorperforming operations, the operations comprising: combining output froma plurality of determination models for an input image of a road, theplurality of determination models each learned using training data inwhich at least one of a state of a moving object mounding an imagingdevice that acquires an image of the road, a state of the road, and anexternal environment is different.
 2. The state determination deviceaccording to claim 1, wherein the operations further comprise:determining and outputting a state of the road based on a result of thecombination by the output combining means.
 3. The state determinationdevice according to claim 1, wherein the operations further comprise:training the plurality of determination models using a plurality ofpieces of training data including images of the road in which at leastone of a state of the moving object, a state of the road, and theexternal environment is mutually different.
 4. The state determinationdevice according to claim 1, wherein the operations further comprise:determining weights to be used for combining output of the plurality ofdetermination models in accordance with an input image of the road; andcombining output of the plurality of determination models using thedetermined weights.
 5. The state determination device according to claim4, wherein the operations further comprise: learning a parameter fordetermining the weights using training data for learning the weights. 6.The state determination device according to claim 5, wherein theoperations further comprise: learning the parameter for determining theweights in accordance with the external environment. 7-8. (canceled) 9.A state determination method comprising: by a state determinationdevice, including combining output from a plurality of determinationmodels for an input image of a road, the plurality of determinationmodels each learned using training data in which at least one of a stateof a moving object mounting an imaging device that acquires an image ofthe road, a state of the road, and an external environment is different.10. The state determination method according to claim 9, furthercomprising: determining a state of the road based on a result of thecombination.
 11. The state determination method according to claim 9,further comprising: training the plurality of determination models usinga plurality of pieces of training data including images of the road inwhich at least one of a state of the moving object, a state of the road,and the external environment is mutually different.
 12. The statedetermination method according to claim 9, further comprising:determining weights to be used for combining output of the plurality ofdetermination models in accordance with an input image of the road; andcombining output of the plurality of determination models using thedetermined weights.
 13. The state determination method according toclaim 12, further comprising: learning a parameter for determining theweights using training data for learning the weights.
 14. The statedetermination method according to claim 13, further comprising: learningthe parameter for determining the weights in accordance with theexternal environment. 15-16. (canceled)
 17. A non-transitorycomputer-readable recording medium embodying a program for causing acomputer to perform a method, the method comprising: combining outputfrom a plurality of determination models for an input image of a road,the plurality of determination models each learned using training datain which at least one of a state of a moving object mounting an imagingdevice that acquires an image of the road, a state of the road, and anexternal environment is different.
 18. The recording medium according toclaim 17, wherein the medium embodies the program for causing thecomputer to perform the method, the method further comprising:determining a state of the road based on a result of the combination.19. The recording medium according to claim 17, wherein the mediumembodies the program for causing the computer to perform the method, themethod further comprising: training the plurality of determinationmodels using a plurality of pieces of training data including images ofthe road in which at least one of a state of the moving object, a stateof the road, and the external environment is mutually different.
 20. Therecording medium according to claim 17, wherein the medium recordsembodies the program for causing the computer to perform the method, themethod further comprising: determining weights to be used for combiningoutput of the plurality of determination models in accordance with aninput image of the road; and combining output of the plurality ofdetermination models using the determined weights.
 21. The recordingmedium according to claim 20, wherein the medium embodies the programfor causing the computer to perform the method, the method furthercomprising: a proccssing of learning a parameter for determining theweights using training data for learning the weights.
 22. The recordingmedium according to claim 21, wherein the medium embodies the programfor causing the computer to perform the method, the method furthercomprising: learning the parameter for determining the weights inaccordance with the external environment.